Estimation of signal distortion using effective sampling density for light field-based free viewpoint video

In a light field-based free viewpoint video (LF-based FVV) system, effective sampling density (ESD) is defined as the number of rays per unit area of the scene that has been acquired and is selected in the rendering process for reconstructing an unknown ray. This paper extends the concept of ESD and shows that ESD is a tractable metric that quantifies the joint impact of the imperfections of LF acquisition and rendering. By deriving and analyzing ESD for the commonly used LF acquisition and rendering methods, it is shown that ESD is an effective indicator determined by system parameters and can be used to directly estimate output video distortion without access to the ground truth. This claim is verified by extensive numerical simulations and comparison to PSNR. Furthermore, an empirical relationship between the output distortion (in PSNR) and the calculated ESD is established to allow direct assessment of the overall video distortion without an actual implementation of the system. A small scale subjective user study is also conducted which indicates a correlation of 0.91 between ESD and perceived quality

3D geometric and haptic modeling of hand-woven textile artifacts

Haptic Modeling of textile has attracted significant interest over the last decade. In spite of extensive research, no generic system has yet been proposed. The majority of the haptic models developed in the previous work assume a 2D mesh model for the textile which does not represent the real geometric configuration of the textile. In addition, they are based on empirical parameters obtained from textile samples using specialized instruments. The process is often time consuming and elaborate, consisting of manual measurement of physical and mechanical properties of the artifacts. The development of a generic approach for 3D haptic modeling of hand-woven textile artifacts is pursued in this work. In the proposed approach, the textile pattern and structure are recognized by digital processing of the artifact still image. A fuzzy-rule based expert system is developed to perform the recognition process. The data obtained in this process is employed to automatically generate the 3D geometric model of the artifact in VRML. The mechanical properties of the artifact are estimated by processing the textile geometric characteristics and yarn properties in a neural network system. These mechanical properties are then deployed in the construction of the textile mechanical model. The mechanical model is superimposed over the 3D geometric model to construct the haptic model. The proposed system is validated through both subjective and objective methods using a number of artifact samples. An extensive review of the published literature on the haptic modeling of textile is provided in the thesis. The benefits of textile haptic modeling are identified. Applications of existing models are reviewed and the significance and unique contribution of the work is presented. The image processing method and the fuzzy rule based expert system deployed in the construction of the geometric model are described in detail. The outcome is a 3D geometric model of the artifact in VRML which could be explored in a virtual reality world viewer. Similarly, the neural network model designed to estimate the mechanical characteristics of an artifact is presented the results of the training and validation of the model are provided. Finally, two methods developed for the haptic model based on geometric and mechanical models in the Reachin are explained. The accuracy and effectiveness of the overall approach are validated through a series of experiments. Overall, the work conduced in this study offers a novel 3D generic haptic modeling for textile artifacts. It can be deployed in museums providing an opportunity for the visitors to touch unique samples of hand-woven textile artifacts. The approach is cost-effective, reliable and reproducible as the haptic modeling of these samples doesn’t need time-consuming and costly laboratory conditions

Effective sampling density for quality assessment and optimization of light field rendering and acquisition

Free Viewpoint Video (FVV) aims to provide users with the ability to select arbitrary views of a dynamic scene in real-time. FVV systems widely adopt simplified plenoptic signal representations, in particular light field (LF). This is referred as an LF-based FVV system in this thesis. An LF-based FVV system consists of three main components: acquisition component, rendering component, and compression/transmission component. The efficacies of these components directly affect the quality of the output video. The main aim of this research is to propose a novel theory and mathematical framework for analytical comparison, evaluation, and optimization of the LF acquisition and rendering components for a realistic under-sampled LF and approximated depth information with errors in depth maps. In contrast, most of the current researches on LF analytical evaluation focus on perfect signal reconstruction and are adequate to objectively predict and assess the influences of imperfections of acquisition and rendering on the output video quality. In the core of the proposed theory there is the concept of effective sampling density (ESD). ESD is shown to be an analytically tractable metric that represents the combined impact of the imperfections of LF acquisition and rendering and can be used to directly predict/estimate output video quality from system parameters. The ESD for the commonly used LF acquisition configurations and rendering methods are derived and analyzed for evaluation and comparison. This claim is verified by extensive numerical simulations. Furthermore, an empirical relationship between the rendering quality (in PSNR) of a system and its ESD is established to allow direct prediction of the overall video quality without the actual implementation of the system. A small scale subjective user study is also conducted which indicates a high correlation between ESD and perceived quality. In addition to comparison and evaluation of LF acquisition and rendering components and objective quality assessment of LF-based FVV systems, ESD theory is also applied to several other significant problems. The first problem is LF acquisition optimization. In particular for a simplified regular grid acquisition, this optimization leads to calculation of the number of cameras required to capture the scene. Existing methods calculate the Nyquist density by assuming a band-limited signal and perfect reconstruction of an arbitrary view using linear interpolation, which often results in an impractically high number of cameras. In contrast, by employing ESD to solve this problem, it is possible to study the problem for under-sampled LF under realistic conditions (non-Lambertian reflections and occlusions) and rendering with complex interpolations. Theoretical and numerical results show that the resulting number of cameras is significantly lower than what was reported in the previous studies with only a few percent reduction in the rendering quality. Moreover, it is shown that the previous methods are special cases of the one derived from ESD theory. The second problem is LF rendering optimization. The ESD theory is utilized to provide an estimation of the rendering complexity in terms of optimum number of rays employed in interpolation algorithm so as to compensate for the adverse effect caused by errors in depth maps for a given rendering quality. The proposed method is particularly useful in designing a rendering algorithm with inaccurate knowledge of depth to achieve the required rendering quality. The third problem is a joint optimization of both LF acquisition and LF rendering to achieve a desired output quality. In particular, the trade-off among acquisition camera density, ray selection, depth error and rendering quality is studied using ESD and methods are presented to optimize these parameters for a system with a desired output quality in terms of ESD or PSNR by applying a Lagrangean method to ESD. Employing the proposed method on a regular grid camera system shows that the number of cameras can be reduced by 8 times if 32 rays, instead of 8 rays, are employed during rendering to achieve a similar rendering quality for a typical 20% error in depth estimation. While in original presentation of ESD, the scene complexity is assumed to be fixed, the fourth problem focuses on the scene complexity and how a non-uniform/irregular acquisition can lead to a higher output quality. LF acquisition is theoretically considered as a problem of plenoptic signal sampling. It is typically performed by using a regular acquisition such as a regular camera grid. While a regular acquisition itself results in non-uniform sampling density, this non-uniformity does not match the scene complexity and frequency variations. To give a solution to the fourth problem the ESD theory is superimposed with the scene complexity and an irregular acquisition method is proposed for optimum non-uniform LF sampling corresponding to the variations of the scene complexity. Specifically, scene complexity is measured through analyzing DCT coefficients of reference images of the scene, describing the frequency behavior of the plenoptic signal over the scene space. An optimization model is formulated to calculate the optimum configurations of the acquisition cameras including positions and orientations. The theoretical analysis and numerical simulations demonstrate that the rendered video quality can be significantly improved (around 20% in mean PSNR) by employing the proposed irregular acquisition compared with the regular camera grid. To validate the proposed theory, a simulation system is proposed. The simulator takes a 3D model of a scene and generates both reference cameras images and ground truth images. The proposed simulation system is highly flexible and efficient to automatically generate different datasets and objectively compare and analyze any LF-based FVV systems for any given experiment design scheme. While the fundamentals of ESD theory is studied and reported in this thesis, the theory requires significant further research. The author is working on extending the ESD theory and applying it to more problems and will report the results in future publications

Optimization of the number of rays in interpolation for light field based free viewpoint systems

Light field (LF) rendering is widely used in free viewpoint video systems (FVV). Different methods have been proposed to employ depth maps to improve the rendering quality. However, estimation of depth is often error-prone. In this paper, a new method based on the concept of effective sampling density (ESD) is proposed for evaluating the depth-based LF rendering algorithms at different levels of errors in the depth estimation. In addition, for a given rendering quality, we provide an estimation of number of rays required in the interpolation algorithm to compensate for the adverse effect caused by errors in depth maps. The proposed method is particularly useful in designing a rendering algorithm with inaccurate knowledge of depth to achieve the required rendering quality. Both the theoretical study and numerical simulations have verified the efficacy of the proposed method

A generic approach to haptic modeling of textile artifacts

Haptic Modeling of textile has attracted significant interest over the last decade. In spite of extensive research, no generic system has been proposed. The previous work mainly assumes that textile has a 2D planar structure. They also require time-consuming measurement of textile properties in construction of the mechanical model. A novel approach for haptic modeling of textile is proposed to overcome the existing shortcomings. The method is generic, assumes a 3D structure for the textile, and deploys computational intelligence to estimate the mechanical properties of textile. The approach is designed primarily for display of textile artifacts in museums. The haptic model is constructed by superimposing the mechanical model of textile over its geometrical model. Digital image processing is applied to the still image of textile to identify its pattern and structure through a fuzzy rule-base algorithm. The 3D geometric model of the artifact is automatically generated in VRML based on the identified pattern and structure obtained from the textile image. Selected mechanical properties of the textile are estimated by an artificial neural network; deploying the textile geometric characteristics and yarn properties as inputs. The estimated mechanical properties are then deployed in the construction of the textile mechanical model. The proposed system is introduced and the developed algorithms are described. The validation of method indicates the feasibility of the approach and its superiority to other haptic modeling algorithms

A quantitative approach for comparison and evaluation of light field rendering techniques

Light field rendering (LFR) is an active research area in computer vision and computer graphics. LFR plays a crucial role in free viewpoint video systems (FW). Although several rendering algorithms have been suggested for LFR but the lack of appropriate datasets with known ground truth has prevented a comparison and evaluation study of LFR algorithms. In most of the LFR papers the method is applied to several test cases for validation and as a result, just a subjective visualized output is given. To overcome this problem, this paper presents a quantitative approach for comparison and evaluation of LFR algorithms. The core of the proposed methodology is a simulation model and a 3D engine. The platform produces the reference images and ground truth data for a given 3D model. Subsequently, data are injected to a comparison engine to compare synthesized images from light field engine with original images from simulation, generating objective results for evaluation. The methodology is highly flexible and efficient to automatically generate LFR datasets and objectively compare and analyze any subset of LFR methods for any given experiment design scheme. Five key rendering algorithms are evaluated with proposed methodology to validate it. Overall, it is shown that the proposed quantitative methodology could be used for LFR objective evaluation and comparison

Annals of Clinical and Translational Neurology

Light field rendering (LFR) is an active research area in computer vision and computer graphics. LFR plays a crucial role in free viewpoint video systems (FVV). Several rendering algorithms have been suggested for LFR. However, comparative evaluation of these methods is often limited to subjective assessment of the output. To overcome this problem, this paper presents a geometric measurement, Effective Sampling Density of the scene, referred to as effective sampling for brevity, for objective comparison and evaluation of LFR algorithms. We have derived the effective sampling for the well-known LFR methods. Both theoretical study and numerical simulation have shown that the proposed effective sampling is an effective indicator of the performance for LFR methods

3D geometric modelling of hand-woven textile

Geometric modeling and haptic rendering of textile has attracted significant interest over the last decade. A haptic representation is created by adding the physical properties of an object to its geometric configuration. While research has been conducted into geometric modeling of fabric, current systems require time-consuming manual recognition of textile specifications and data entry. The development of a generic approach for construction of the 3D geometric model of a woven textile is pursued in this work. The geometric model would be superimposed by a haptic model in the future work. The focus at this stage is on hand-woven textile artifacts for display in museums. A fuzzy rule based algorithm is applied to the still images of the artifacts to generate the 3D model. The derived model is exported as a 3D VRML model of the textile for visual representation and haptic rendering. An overview of the approach is provided and the developed algorithm is described. The approach is validated by applying the algorithm to different textile samples and comparing the produced models with the actual structure and pattern of the samples

A method for calculating the minimum number of cameras in a light field based free viewpoint video system

Calculation of the number of cameras required to capture the scene is an essential problem in a practical light field based free viewpoint video (FVV) system. Existing methods calculate the Nyquist rate by assuming a band-limited signal and perfect reconstruction of an arbitrary view using linear interpolation, which often results in an impractically high number of cameras. This paper proposes a new method based on the concept of effective sampling density (ESD). It is demonstrated that there is a trade-off between the depth information accuracy, the required number of cameras, and the desired rendering quality, which could be exploited to minimize the number of cameras for a given objective. Theoretical and numerical results show that the resulting number of cameras is significantly lower than what was reported in other studies with only a few percent reduction in the output PSNR. Moreover, it is shown that the methods proposed in those studies are special cases of the one presented in this paper